Segmentation using Deformable Spatial Priors with Application to Clothing

The formulation of image segmentation as maximum a posteriori probability (MAP) inference over a Markov Random Field (MRF) is both elegant and effective. Typically, the MRF is configured to favour contiguous regions with the same labelling, and consistency between the label at each pixel and prior intensity distributions for foreground and background regions. Boykov and Jolly show how to solve this labelling problem efficiently by reformulating as finding a minimum graph-cut [1]. In an extension to this method, the colour distribution is treated as a latent property [5] and in a further extension (OBJ CUT) [3], the foreground object is assumed to be an instance of a known object category, so that prior shape information can be exploited as a top-down influence. A 2D spatial prior on the foreground/background probabilities for each pixel is used as shape model in [4] and with smooth deformations in [6]. Our novel contribution is threefold: (1) we use a spatial prior with category specific deformation function, ranging over multiple labels corresponding to the different parts of an object; (2) we deal jointly with multiple overlapping object instances within the same image, integrating this into a global optimisation within the same MRF framework; (3) we demonstrate an improvement using this approach on the state of the art for the problem of clothing segmentation from images of groups of people. The use of a deformable prior is motivated by the Active Appearance Model (AAM) [2], except that we deform a map of prior probabilities for the labels at each location, relative to an object-centred frame of reference, rather than textures. The input to our method is an image D = {d1,d2, . . . ,dN}, where di is the observed RGB colour at pixel i, and J instance hypotheses {o1,o2, . . . , oJ} for a known object category, specifying their position and scale in the image. A candidate segmentation is an assignment of one of K +1 labels to each pixel in the input image L = {l1, l2, . . . , lN}, meaning that a pixel belongs to one of K object parts or background. The segmentation task is posed as inference in a MRF model. Given prior information about the shape of each object S = {s1,s2, . . . ,sJ} and the colour model for each part of each object denoted by Θ = {θ11, . . . ,θ1K ,θ21, . . . ,θ2k, . . . ,θJ1, . . . ,θJK}, we seek the solution L̂ which maximises the posterior probability for L: